fix(recipes): raise K8s floors to clear the DRA chart's kubeVersion - #2449

Summary

Raises the K8s.server.version floor to >= 1.32 on the 29 overlays that declared a lower one, so no recipe admits a cluster the pinned NVIDIA DRA driver chart will refuse to install on. Adds a guard so the reconciliation cannot drift back.

Fixes: #2402

Motivation / Context

Every recipe carries a DRA driver. base.yaml declares nvidia-dra-driver-gpu, and the one overlay that disables it — recipes/overlays/ocp.yaml — substitutes nvidia-dra-driver-gpu-ocp in its place. Both resolve to the same upstream chart, which declares:

kubeVersion: ">=1.32.0-0"

Helm refuses the install below that. So 29 overlays declaring a lower floor admitted clusters that pass every recipe-time check and then fail at helm install. The recipe validates clean and the deploy breaks, which is the worst place for it to surface.

This was already diagnosed once and never generalized. recipes/overlays/ocp.yaml carries >= 1.32 with the comment:

"Raised from >= 1.29 so DRA's floor doesn't silently pass recipe-time constraint checks and then fail at helm install."

Same chart, same reasoning, one family. This reconciles the rest of the catalog.

Related: #2438 (same missing-requirement-expression pattern, driver-version instance), #2439 (GPU stack bump — see Sequencing)

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature
• Breaking change
• Documentation update
• Refactoring (no functional changes)
• Build/CI/tooling

Component(s) Affected

• CLI (cmd/aicr, pkg/cli)
• API server
• Recipe engine / data (pkg/recipe)
• Bundlers
• Collectors / snapshotter
• Validator
• Core libraries (pkg/defaults comment correction)
• Docs/examples

Implementation Notes

Why every declaration, not just base.yaml

Raising the base floor alone would not have worked. Constraints merge by name with the later overlay winning and no max comparison — RecipeMetadataSpec.Merge in pkg/recipe/metadata.go for top-level spec.constraints, mergeValidationPhase in pkg/recipe/validation.go for validation-phase constraints — so a leaf declaring ">= 1.30" silently overwrites a higher floor inherited from base.

That is the same last-wins hazard documented for driver floors in #2438, showing up in a second issue — it is a general property of the constraint system, not a quirk of one constraint name.

The golden churn demonstrates it. Exactly 16 of 48 leaves moved: those inheriting a raised floor, including every *-any wildcard. The leaves that did not move declare their own >= 1.34 or >= 1.32.4, which already cleared 1.32 and were overwriting base's value anyway — so their effective constraint is unchanged.

Overlays raised

29 declarations, all from a sub-1.32 value to >= 1.32:

The guard

TestOverlayK8sFloorsClearDRAChartFloor asserts that no overlay or mixin declares a floor which admits a cluster below the chart's kubeVersion. It verified 95 declarations on this branch.

It does not read the expression — it proves a lower bound on it. Each overlay and mixin is decoded into the typed recipe.RecipeMetadata, and every K8s.server.version constraint found in spec.constraints, in any validation phase, and in any profile value is parsed with the shipping parser (constraints.ParseCompoundConstraint). proveExpressionClearsFloor then walks the parsed structure and proves its effective lower bound symbolically.

Two set-theoretic facts carry the proof. An AND group's satisfying set is the intersection of its terms, so the group clears the floor as soon as any one term does. A compound's satisfying set is the union of its groups, so every group must clear it — one loose alternative admits a sub-floor cluster regardless of its siblings. Only >=, >, == and a bare exact version place a lower bound; <, <= and != place none.

Symbolic proof rather than sampling is the point. An earlier version of this guard evaluated each expression against a fixed list of probe versions, which cannot cover the grammar: ">= 1.32 || > 1.31.0 < 1.31.2" is valid catalog syntax, is accepted by the production evaluator for Kubernetes 1.31.1, and passed the sampled guard. Typed decoding likewise removes YAML layout — key order, quoting style, comments, indentation — from the picture entirely.

The guard fails closed on anything it cannot reason about: an unparseable value, major-only precision (>= 1), an unknown operator, zero alternatives, or an empty group. It also fails when zero declarations match, so it cannot go silently inert. One deliberate conservatism is documented in code: for > v the prover requires v itself to clear the floor, which is one patch stricter than strictly necessary but never fails open, since the patch component is unbounded.

Coupling to the registry. TestDRAChartFloorAuditIsCurrent holds an audit table recording, per DRA component, the chart version whose kubeVersion was actually read. Both catalog DRA components are enrolled — OCP disables the generic nvidia-dra-driver-gpu and substitutes nvidia-dra-driver-gpu-ocp, so covering only the generic one would leave the OCP chain unguarded. If a pin in registry.yaml moves away from the audited version, the test fails until someone re-reads the chart's kubeVersion and updates the table. That is the point: the guard cannot sit green at a stale floor after a chart bump.

Controls demonstrated. Each was applied to recipes/overlays/base.yaml, the guard was run, and the probe reverted; git status --porcelain is clean afterwards.

The last is the shape that defeated the previous sampled version of this guard, and is now a permanent row in TestProveExpressionClearsFloor. TestProveExpressionRejectsWhatTheEvaluatorAdmits is an adversarial companion: it independently asserts the production evaluator really does accept 1.31.1 for that expression. Without it, a prover bug that rejected everything would leave the table green — a vacuous pass.

Typed decoding also surfaced one declaration the earlier text scan had never counted — 95 rather than 94.

What this does not do

The alternative from #2402 — making the nvidia-dra-driver-gpu componentRef conditional on server version so low-floor overlays omit it — is not implemented. It is more invasive and would leave those overlays without ComputeDomain/IMEX. It would only be worth revisiting if some family genuinely needs to support sub-1.32 clusters, which nothing in the catalog currently asserts.

Sequencing

Merge #2439 first, then this, then #2446. All three regenerate the same two parity-golden files and will conflict pairwise. Resolve by rebasing and regenerating — these are derived files, and hand-picking a conflict side produces a golden matching neither tree.

Testing

make qualify
golangci-lint run -c .golangci.yaml ./pkg/recipe/... ./pkg/defaults/...
go test -race ./pkg/recipe/... ./pkg/defaults/...
AICR_UPDATE_GOLDEN=1 go test ./pkg/recipe/... -run TestCatalogParityGolden
AICR_UPDATE_GOLDEN=1 go test ./pkg/bundler/... -run TestStockRenderParityGolden

golangci-lint on ./pkg/recipe/... and ./pkg/defaults/... reports 0 issues., captured to a file and gated on the exit code rather than piped.

Golden regeneration was needed only for the constraint-value changes. The goldens are leaf: sha256 lines only — no structural changes — and the 16 moved leaves are fully accounted for above. The later guard rework is test-only and the comment corrections are comments, so neither moved the goldens; that was confirmed rather than assumed by re-running both parity tests without AICR_UPDATE_GOLDEN.

The seven guard controls are in the Implementation Notes table above. Each was applied to the working tree, run, and reverted, and the tree was confirmed clean afterwards.

make qualify passes on the current head. Verified by reading the log rather than trusting the exit code: 9,774 lines, zero --- FAIL lines, the Codebase qualification completed sentinel present, and QUALIFY_EXIT=0.

Note for reproduction: make qualify cannot complete inside a restricted sandbox — tools/api-diff_test.sh fails with mktemp: Operation not permitted, and overriding TMPDIR does not help because the script resets it. The passing run was unsandboxed. A sandboxed attempt was also observed to report exit code 0 while its log showed failures, which is why the verdict above is quoted from the log rather than the exit status.

Risk Assessment

• Low
• Medium — Touches multiple components or has broader impact
• High

The change itself is 29 constraint values and one test. What makes it Medium is the user-visible effect: a cluster on Kubernetes 1.30 or 1.31 that previously resolved a recipe will now be rejected at generation.

That is the intended fix, not a regression — such a cluster was already broken, it just failed later and less legibly, at helm install. Nobody loses a working configuration; they lose a configuration that only appeared to work until deploy time.

Rollout notes: No cluster-side action. Anyone pinned below 1.32 who relied on recipe generation succeeding will now get a clear constraint failure naming the required version instead of a Helm chart requires kubeVersion error mid-deploy.

Checklist

• Tests pass locally (make test with -race)
• Linter passes (make lint) — golangci-lint on ./pkg/recipe/... and ./pkg/defaults/... reports 0 issues
• I did not skip/disable tests to make CI green
• I added/updated tests for new functionality
• I updated docs if user-facing behavior changed — the user-facing docs regenerated with the floor bump are in the diff (docs/user/cli-reference.md, docs/user/component-catalog.md, demos/query.md), and two stale in-code references were corrected: the MirrorDefaultKubeVersion note that still named the old >= 1.25 base floor, and the A100 GKE comment that contrasted its own floor with an H100 floor it now shares
• Changes follow existing patterns in the codebase
• Commits are cryptographically signed (git commit -S)